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Background / Context: 

Due to the lack of effectiveness of the punitive school approach toward challenging 
behaviors (Luiselli, Putnam, Handler, & Feinberg, 2005; Reynolds, Skiba, Graham, Sheras, 
Conoley, & Garcia- Vazquez, 2006), public schools have searched for an innovative approach to 
better serve students who are at risk for academic failure and dropout/expulsion. A growing 
number of schools have considered School-Wide Positive Behavior Intervention Support (SW- 
PBIS) as an innovative approach that sheds light on prevention rather than on disciplinary 
enforcement. As emphasis on student achievement and social competency grows, SW-PBIS has 
been introduced as a whole-school approach to develop and enhance student social competency 
and learning (Sugai, 2007) 

SW-PBIS provides schools with effective systemic support for challenging behaviors. It 
is characterized as a value added model that involves the integration of value-added outcomes; 
data-based decision making; human ecology and behavior science; scientific evidence-based 
practices; and school system changes (Anderson & Kincaid, 2005; Carr et al., 1999, 2002; Center 
on Positive Behavioral Interventions and Supports, 2004; Lewis & Sugai, 1999; Safiran & 
Oswald, 2003; Sugai, Homer, and Gresham, 2001; Sugai et al., 2000; Utley & Sailor, 2002, as 
cited in Lafrance 2009). Lafrance's study (2009) summarized that SW-PBIS outcomes include 
decreased numbers of office discipline referrals (ODR); improved prosocial behaviors and peer 
relationships; and increased instructional time, academic achievement, and satisfaction about 
school safety. 

Schools implementing SW-PBIS desire empirical evidence that indicates a linkage 
between SW-PBIS and student learning. However, the few studies that have been conducted 
have yielded mixed results. Sailor and colleagues’ (2006) concluded that three middle schools 
and one K-3 charter school implementing SW-PBIS showed improved school-level academic 
achievement. New Hampshire’s two-year study also reported a positive relationship between 
SW-PBIS and student achievement in math (Muscott, Mann, and LeBrun, 2008). In contrast, 
Bradshaw, Mitchell, and Leaf (2009) reported that no relationship between SW-PBIS and 
academic achievement was identified when examining the effect of SW-PBIS on Maryland’s 
thirty seven elementary schools’ academic performance. Taken together, it appears the 
effectiveness of SW-PBIS on student achievement at the school level is inconclusive; thus, 
research efforts to examine the longitudinal effect of SW-PBIS on academic achievement growth 
are needed. 

Purpose / Objective / Research Question / Focus of Study: 

Although there are some studies investigating the effectiveness of SW-PBIS on students’ 
or schools’ accountability, no studies investigate the effect of SW-PBIS over different grade 
levels with statewide standardized tests. Our study purports to investigate the effect at both 
elementary and middle schools with 3 -year longitudinal statewide test data. In line with the 
purpose, three research questions guided this research: (1) Is SW-PBIS effective on schools’ 
accountability as represented by statewide standardized tests; (2) What is the pattern of the effect 
on schools’ accountability over time?; and (3) Which school characteristics or factors boost the 
effect on school’s accountability? 
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Setting: 

Minnesota SW-PBIS (MN SW-PBIS) is a large-scale implementation in Minnesota. MN 
SW-PBIS provides all necessary training and technical support to improve fidelity of 
implementation of SW-PBIS and has been implemented by a total of 139 out of 2006 public 
schools (MDE, 2011). This study examined the longitudinal effects of Minnesota SW-PBIS on 
school-level student achievement. 

Population / Participants / Subjects: 

Participating schools consist of two groups: one includes elementary schools having 
3 ld to 5 th grades and the other includes middle schools having 6 th to 8 th grades between the 
years 2008 and 2010. Since a preliminary study on elementary schools is just completed, we 
only consider the elementary schools in this abstract. Sample data include schools that start 
from 3 rd grade at 2008 to 5 th grade at 2010 and were selected from the state of Minnesota. In 
Minnesota, the SW-PBIS program started in 2005 and the number of schools implementing 
SW-PBIS has steadily increased over time. Elementary schools that implemented SW-PBIS 
program are summarized in Table 1. 

(Please insert Table 1 here) 

According to the first year of implementation, three cohort groups, cohorts 3 to 5 in 
Table 1, were selected for this study. Cohorts 3 to 5 consist of schools that started 
implementing SW-PBIS in 2008, 2009, and 2010, respectively. SW-PBIS intervention 
schools are considered as a treatment group while matched schools to each treatment group 
are selected by applying a propensity matching method. The propensity matching method 
will be discussed in detail in the Statistical Model section. Thus, the sample consists of 
three cohort groups and their matched schools; the sample size after matching is 144 
(=2*(17+1 1+29)). All schools are public and participated in statewide achievement tests, the 
Minnesota Comprehensive Assessment - II (MCA II) in math and reading. By applying the 
propensity matching method, we are able to reduce selection bias and draw results from a 
quasi-experimental design. 

Significance / Novelty of study: 

Description of what is missing in previous work and the contribution the study makes. 

This study is innovative because we investigated the effectiveness of SW-PBIS on 
statewide standardized tests in large-scale longitudinal data at the both elementary and middle 
school levels. Identification of factors listed in the third research question is a very significant 
study in the field of positive behavior in schools. Furthermore, our results will provide rationales 
for evaluators and policy makers related to the SW-PBIS program. 

This evidence-based research on SW-PBIS provides broad views of current affairs in 
SW-PBIS. To increase validity of this study, we applied novel methodologies such as propensity 
score matching methods (PSMM), multivariate linear mixed models (MLMM), and longitudinal 
structural equation modeling (LSEM). First, we applied a PSMM resulting in a quasi- 
experimental research design. Based on matched data, we used advanced methodologies, 

MLMM and LSEM, to draw statistical inferences in this study. 
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Statistical, Measurement, or Econometric Model: 

Propensity Score Matching Method: 

We applied a PSMM to select non SW-PBIS schools that reduce sampling bias. 
Among the matching methods, such as exact, subclassification, nearest neighbor, optimal, 
full, generic, and coarsened exact matching (Ho, Imai, King, and Stuart, 2011), we applied 
-optimal” matching that is fairly similar as -nearest neighbor” matching using propensity 
scores obtained by logistic regression. Optimal matching finds matched samples to 
minimize a global distance measure that is different from the nearest neighbor matching. 

The efficiency was shown by Gu and Rosenbaum (1993). To conduct this analysis, we used 
the Mat chit package in R. 

Multivariate Linear Mixed Model for Longitudinal Data: 

Longitudinal data obtained by PSMM consists of multiple-responses, including 
mathematics and reading test scores, over 3 years. In analysis, the correlation between 
mathematics and reading test scores will be considered by applying MLMM. This analysis will 
provide the effectiveness of SW-PBIS on schools’ accountability represented by both 
mathematics and reading. Using matrix notation, MLMM can be formulated as 

Y ik | b ik ~ N(X ik P k + Z, A ,£,*), k = 1 and 2 

where N( y) denotes a multidimensional normal distribution, X ik and Z ik are design matrices 
for the fixed and subject-specific effects, respectively, and [f. is a vector of unknown regression 
coefficients (Fitzmaurice, Davidian, Verbeke, and Molenberghs, 2009). To conduct this analysis, 
we use the SabreR package in R. 

Longitudinal Structure Equation Modeling (LSEM): 

As indicated in the third research question, in this study we identify significant factors 
(school characteristics) that boost the effect of SW-PBIS. To avoid the redundancy of using 
covariates at both PSMM and LSEM, we use raw data sets including all covariates used in 
PSMM, the SW-PBIS variable, and math and reading test scores. By investigating the mediator 
or moderator effects of school characteristics over SW-PBIS on schools’ accountability, we draw 
school characteristics correlated with SW-PBIS program on school’s accountability. To conduct 
this analysis, we use the OpenMx package in R. 

Usefulness / Applicability of Method: 

In educational and behavioral research, PSMM is very useful to set up a quasi- 
experimental design. Raw data available to us were observational data consisting of all SW-PBIS 
schools and non SW-PBIS schools, which may cause a selection bias. This potential problem 
results in inaccurate statistical inference. As indicated, we selected a matched sample from 
each cohort group according to the year of implementation of SW-PBIS. Thus, we have 
three different results from three matching procedures. Tables 2 to 4 show the improvement 
of balance by comparing statistics before and after matching. For example, -Distance” 
measuring a global distance was decreased as 62.36%, 98.55%, and 99.32% at cohort 
groups. In addition to statistics, Figures 1 to 3 indicate comparison of distribution of 
propensity scores. Points under Matched Treatment Units and Matched Control Units are 



SREE Fall 2011 Conference Abstract Template 



3 




data selected by PSMM. Regardless of distributions of the raw data, each figure shows that 
the distribution of the matched sample is fairly similar to that of the treatment sample. This 
procedure removes the threat of selection bias. 

Two advanced methodologies, MLMM and LSEM, enable us to answer research 
questions without correcting due to the unavailability of methodology. Likewise PSMM, 
MLMM and LSEM have strengths in drawing accurate results. For example, MLMM deals 
with potential correlation between outcomes while LSEM provides the flexibility to test 
models under various conditions (e.g., -what if...?” types of questions). All methodologies, 
PSMM, MLMM, and LSEM, can be applicable in social and behavioral research. We hope 
through this project to be able to provide advanced methodologies in the analysis of 
educational and psychology data. 

Research Design: 

This project is a quasi-experimental design using a matched sample that is obtained by a 
PSMM. This project is also both a longitudinal study with 3-year longitudinal data and a multi- 
group study with elementary and middle school data. Since professional development and 
coaching for SW-PBIS are provided during the summer and MCA II is administered at the end of 
spring semester, it is expected that degree of awareness of positive behavior in school differs 
from the length of SW-PBIS program in school. Thus, a longitudinal study is more appropriate 
than a cross-sectional study to investigate the effectiveness of SW-PBIS. It is beyond the scope 
to discuss the relations among awareness, length, and accountability, but the potential 
relationship cannot be avoided. 

Data Collection and Analysis: 

Based on the matched sample consisting of 57 SW-PBIS and 57 non SW-PBIS schools in 
elementary school, we investigated the effect of SW-PBIS on MCA II math and reading scores, 
separately. By fitting LMMs into 3 year longitudinal data, we had base models to test the effect 
of SW-PBIS. When selecting and testing models, we applied the delta method based on BIC 
(Kadane and Lazar, 2004). 

Findings / Results: 

Preliminary data analysis based on elementary schools indicates no statistically 
significant relationship between SW-PBIS program and schools’ accountability. Interestingly, 
the matched sample still indicates a fairly large gap on schools’ accountability between cohort 
groups and matched groups (Figures 4 to 9). 

Conclusions: 

Although our preliminary data analysis based on separate linear mixed models indicates 
no significant effect of SW-PBIS, further analyses are planned in order to investigate this issue 
more fully. 
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Appendix B. Tables and Figures 

Table 1 

Cumulative List of Elementary Schools 



First year of implementation 


Number of schools 


Selected sample 


Cohort 1 (2005-2006) 


4 




Cohort 2 (2006-2007) 


10 




Cohort 3 (2007-2008) 


17 


17 


Cohort 4 (2008-2009) 


11 


11 


Cohort 5 (2009-2010) 


29 


29 


Total 


71 


57 
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Table 2 



Result of Propensity Score Matching at Cohort 3 









All data 










Matched data 






%of 


Variables 


Means 




SD 




Mean 


Means 




SD 




Mean 


Improvement 




SW- 


Non SW- 


Non SW- 


Diff 


SW- 


Non SW- 


Non SW- 


Diff 






PBIS 


PBIS 




PBIS 






PBIS 


PBIS 




PBIS 








distance 


0.26 




0.02 




0.04 


0.25 


0.26 




0.17 




0.18 


0.09 


62.36 


School size 


66.41 




64.07 




39.28 


2.34 


66.41 




55.00 




62.21 


11.41 


-387.61 


American Indian 


2.47 




1.36 




4.68 


1.11 


2.47 




0.88 




1.50 


1.59 


-42.47 


Asia/Pacific Islander 


1.53 




2.98 




5.61 


-1.45 


1.53 




0.47 




0.87 


1.06 


26.82 


Hispanic 


2.00 




2.98 




5.24 


-0.98 


2.00 




1.18 




2.01 


0.82 


15.98 


Black 


17.88 




5.51 




8.62 


12.37 


17.88 




11.29 




18.22 


6.59 


46.74 


White 


42.47 




51.23 




37.81 


-8.76 


42.47 




41.06 




64.35 


1.41 


83.88 


Male 


34.12 




32.69 




20.78 


1.43 


34.12 




27.94 




33.99 


6.18 


-333.18 


Special Education 
Serv. 


9.35 




8.28 




5.78 


1.08 


9.35 




6.53 




5.81 


2.82 


-161.97 


Limited Eng. 
Proficiency 


1.35 




2.57 




7.17 


-1.21 


1.35 




0.53 




1.37 


0.82 


32.12 


Free/Reduced Priced 
Meals 


30.76 




20.31 




14.73 


10.46 


30.76 




24.82 




20.01 


5.94 


43.18 


Migrant Service 


0.00 




0.08 




0.52 


-0.08 


0.00 




0.00 




0.00 


0.00 


100.00 


Percent of Level 1 (M) 


19.74 




5.79 




6.84 


13.94 


19.74 




15.67 




13.57 


4.06 


70.85 


Percent of Level 2 (M) 


24.22 




16.29 




9.47 


7.93 


24.22 




27.63 




10.41 


-3.41 


57.04 


Percent of Level 3 (M) 


39.66 




45.40 




10.51 


-5.74 


39.66 




39.94 




13.50 


-0.28 


95.18 


Percent of Level 4 (M ) 


16.39 




32.52 




15.93 


-16.13 


16.39 




16.79 




14.66 


-0.39 


97.56 


MCA II Math (2007) 


351.25 




358.22 




5.00 


-6.96 


351.25 




352.06 




5.71 


-0.81 


88.34 


Percent of Level 1 (R) 


26.51 




11.34 




10.17 


15.17 


26.51 




25.14 




17.93 


1.38 


90.93 


Percent of Level 2 (R) 


10.53 




9.52 




5.73 


1.01 


10.53 




9.82 




6.75 


0.71 


29.74 


Percent of Level 3 (R) 


32.93 




31.21 




8.94 


1.72 


32.93 




33.94 




9.98 


-1.01 


41.35 


Percent of Level 4 (R) 


30.03 




47.93 




16.37 


-17.90 


30.03 




31.13 




20.72 


-1.10 


93.86 


MCA II Reading 
(2007) 


353.10 




362.81 




7.37 


-9.71 


353.10 




353.78 




11.04 


-0.68 


92.98 
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Table 3 



Result of Propensity Score Matching at Cohort 4 









All data 










Matched data 






%of 


Variables 


Means 




SD 




Mean 


Means 




SD 




Mean 


Improvement 




SW- 


Non SW- 


Non SW- 


Diff 


SW- 


Non SW- 


Non SW- 


Diff 






PBIS 


PBIS 




PBIS 






PBIS 


PBIS 




PBIS 








distance 


0.08 




0.01 




0.04 


0.07 


0.08 




0.08 




0.06 


0.00 


98.55 


School size 


85.91 




64.28 




38.64 


21.63 


85.91 




92.27 




49.70 


-6.36 


70.58 


American Indian 


1.27 




1.37 




4.73 


-0.09 


1.27 




2.55 




2.88 


-1.27 


-1254.19 


Asia/Pacific Islander 


3.36 




3.03 




5.66 


0.33 


3.36 




3.73 




4.63 


-0.36 


-10.26 


Hispanic 


6.09 




3.02 




5.29 


3.07 


6.09 




9.36 




9.70 


-3.27 


-6.63 


Black 


7.18 




5.38 




8.24 


1.80 


7.18 




13.00 




16.35 


-5.82 


-222.71 


White 


67.82 




51.46 




37.01 


16.36 


67.82 




63.64 




45.06 


4.18 


74.44 


Male 


44.36 




32.80 




20.40 


11.56 


44.36 




49.55 




27.62 


-5.18 


55.18 


Special Education 
Serv. 


14.27 




8.32 




5.77 


5.96 


14.27 




14.36 




9.67 


-0.09 


98.47 


Limited Eng. 
Proficiency 


5.73 




2.61 




7.24 


3.11 


5.73 




8.27 




10.48 


-2.55 


18.27 


Free/Reduced Priced 
Meals 


28.55 




20.20 




14.59 


8.34 


28.55 




38.27 




23.68 


-9.73 


-16.62 


Migrant Service 


0.18 




0.09 




0.52 


0.10 


0.18 




0.64 




1.21 


-0.45 


-370.70 


Percent of Level 1 (M) 


6.85 




5.57 




6.45 


1.29 


6.85 




10.55 




9.42 


-3.70 


-186.94 


Percent of Level 2 (M) 


18.87 




16.03 




9.29 


2.84 


18.87 




25.36 




11.17 


-6.49 


-128.65 


Percent of Level 3 (M ) 


43.93 




45.52 




10.41 


-1.60 


43.93 




42.12 




8.83 


1.81 


-13.36 


Percent of Level 4 (M ) 


30.35 




32.88 




15.78 


-2.53 


30.35 




21.98 




12.97 


8.37 


-231.18 


MCA II Math (2007) 


357.90 




358.36 




4.90 


-0.46 


357.90 




354.60 




4.98 


3.30 


-621.08 


Percent of Level 1 (R) 


12.92 




11.03 




9.71 


1.89 


12.92 




16.09 




8.93 


-3.17 


-67.64 


Percent of Level 2 (R) 


9.91 




9.51 




5.71 


0.40 


9.91 




10.62 




4.37 


-0.71 


-77.44 


Percent of Level 3 (R) 


32.66 




31.15 




8.91 


1.51 


32.66 




36.55 




9.64 


-3.89 


-157.37 


Percent of Level 4 (R) 


44.53 




48.32 




16.07 


-3.79 


44.53 




36.76 




11.40 


7.76 


-104.87 


MCA II Reading 
(2007) 


362.41 




363.02 




7.14 


-0.61 


362.41 




359.19 




5.29 


3.22 


-425.37 
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Table 4 



Result of Propensity Score Matching at Cohort 5 









All data 










Matched data 






%of 


Variables 


Means 




SD 




Mean 


Means 




SD 




Mean 


Improvement 




SW- 

PBIS 


Non SW- 
PBIS 


Non SW- 
PBIS 


Diff 


SW- 

PBIS 


Non SW- 
PBIS 


Non SW- 
PBIS 


Diff 




distance 


0.08 




0.04 




0.04 


0.04 


0.08 




0.08 




0.06 


0.00 


99.32 


School size 


65.69 




63.86 




38.33 


1.83 


65.69 




43.21 




29.13 


22.48 


-1126.88 


American Indian 


1.83 




1.35 




4.75 


0.48 


1.83 




3.83 




15.09 


-2.00 


-317.81 


Asia/Pacific Islander 


3.79 




3.02 




5.68 


0.77 


3.79 




3.17 




9.85 


0.62 


19.37 


Hispanic 


4.24 




2.93 




5.14 


1.32 


4.24 




3.48 




9.09 


0.76 


42.33 


Black 


4.10 




5.26 




8.03 


-1.16 


4.10 




1.55 




4.81 


2.55 


-119.92 


White 


51.72 




51.28 




36.89 


0.45 


51.72 




31.17 




25.80 


20.55 


-4487.04 


Male 


34.59 




32.55 




20.19 


2.04 


34.59 




22.97 




15.73 


11.62 


-470.16 


Special Education 
Serv. 


9.97 




8.22 




5.65 


1.74 


9.97 




6.62 




5.22 


3.34 


-92.05 


Limited Eng. 
Proficiency 


3.00 




2.53 




7.16 


0.47 


3.00 




4.48 




13.65 


-1.48 


-213.79 


Free/Reduced Priced 
Meals 


23.79 




19.93 




14.26 


3.86 


23.79 




20.31 




20.44 


3.48 


9.81 


Migrant Service 


0.07 




0.08 




0.50 


-0.01 


0.07 




0.28 




1.13 


-0.21 


-2500.00 


Percent of Level 1 (M) 


5.55 




5.49 




6.37 


0.06 


5.55 




6.77 




6.99 


-1.22 


-1977.98 


Percent of Level 2 (M) 


16.59 




15.89 




9.20 


0.70 


16.59 




20.22 




9.85 


-3.63 


-421.70 


Percent of Level 3 (M) 


49.25 




45.57 




10.42 


3.67 


49.25 




50.41 




11.41 


-1.16 


68.37 


Percent of Level 4 (M) 


28.60 




33.05 




15.77 


-4.45 


28.60 




22.58 




18.42 


6.02 


-35.19 


MCA II Math (2007) 


357.48 




358.41 




4.88 


-0.93 


357.48 




355.82 




5.70 


1.67 


-78.77 


Percent of Level 1 (R) 


10.44 




10.95 




9.71 


-0.51 


10.44 




13.37 




13.85 


-2.93 


-474.14 


Percent of Level 2 (R) 


9.87 




9.49 




5.73 


0.38 


9.87 




11.08 




7.61 


-1.21 


-220.77 


Percent of Level 3 (R) 


32.69 




31.07 




8.88 


1.62 


32.69 




33.02 




11.44 


-0.33 


79.60 


Percent of Level 4 (R) 


47.01 




48.49 




16.07 


-1.48 


47.01 




42.51 




20.12 


4.49 


-202.68 


MCA II Reading 
(2007) 


362.93 




363.08 




7.15 


-0.15 


362.93 




360.97 




9.79 


1.96 


-1218.95 
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Figure 1 

Comparison between treatment and control 3 schools via distribution of propensity scores 
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Figure 2 

Comparison between treatment and control 4 schools via distribution of propensity scores 
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Figure 3 

Comparison between treatment and control 5 schools via distribution of propensity scores 



Distribution of Propensity Scores 



Unmatched Treatment Units 




Matched Treatment Units 




o 

1 

& 

O 

o 




Matched Control Units 




go ° 

SP o o#o ° ° o 




Unmatched Control Units 




o 

o 

^fo 


o 



0.0 0.1 0.2 0.3 0.4 0.5 0.6 

Propensity Score 



SREE Fall 2011 Conference Abstract Template 



B-l 




MCA II - Mathematics 
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